Since they were introduced and started to be largely adopted in the 70's databases have proliferated in all sorts of domains including engineering, scientific, commercial and business applications. Their size can be anything ranging from a small database used by a single individual on a personal computer, e.g., to keep track of personal finances, to large and very large databases set up by various institutions, companies and commercial organizations to support their activity. In an all-interconnected world those large databases are also generally, if not always, made accessible to numerous remotely located end-users to query whatever information is made available by the databases.
In the airline industry, examples of such very-large databases are the ones that hold the airline fares along with the rules restricting their use. Fare databases are mainly set up by a few worldwide global distribution systems (GDSs) that provide travel services to actors of the travel industry including the traditional travel agencies and all sorts of other online travel service providers. Those large databases must generally be operational in a 24-hour-a-day/7-day-a-week mode to sustain a worldwide business that never sleeps while they also need to constantly acquire new fares published by hundreds of large and smaller airline companies. Huge volume of airfares data to be integrated into the database is received daily. The data received are variable and unpredictable in term of number of files, volume (from 0 to millions of records) and functional content (fares, rules, routings . . . ) and they are not filed the same way according to their provider.
The current trend is an increase both of the volume of each transmission and of the frequency. For instance ATPCo (which stands for Airline Tariff Publishing Company, a historical fare provider) have announced that they have sent hourly transmissions in 2010, instead of 10 times a day previously, more than doubling the frequency of their previous sending.
Fare definitions are usually made of several components comprising Fares (general data with fare amounts), Rules (which specify criteria applicable to the fares) and routings (typically ordered lists of intermediary cities through which a trip from an origin to a destination can be made).
New fare definitions are usually provided by the provider in the form of files which need to be processed by a computer system before a loading stage when the new fares, then stored in a database, are made available to a production system which is by way of example a portion of a computerized reservation system handling requests of end users such as travelers or travel agents in the perspective of returning information on travel solutions.
Current techniques for processing new fare definitions to be loaded in database involve fixed computer resources. Such resources are usually oversized to respect as often as possible a maximum processing time set up in a service level agreement (SLA) between the travel company (typically an airline) and the computer service provider (such as a GDS); but in case of peak period of fare filing, the SLA is even not fulfilled: an alert is then raised, requiring an immediate action.
Hence, there is a need for an improved technique for processing data to be loaded in database to optimize the resource consumption in every situation even when the volume of data to be processed varies in large proportions.